Fast convergence of evpn networks for multi homing topologies

ABSTRACT

In general, techniques of this disclosure may enable a remote PE router to improve convergence time in response to a link failure in an EVPN by establishing per-Ethernet Segment Identifier (ESI) Bidirectional Forwarding Detection (BFD) sessions with other PE routers that are coupled to the PE router in an EVPN. The remote PE may determine that at least two PE routers with the remote PE are locally connected to a multi-homed customer network by a particular Ethernet Segment. The remote PE may send, based on determining that the at least two PE routers are connected to the multi-homed customer network by the particular Ethernet Segment, an Ethernet Segment Identifier (ESI) Ping request packet through the intermediate network to one of the at least two PE routers, wherein the ESI Ping request packet includes at least a Bidirectional Forwarding Detection (BFD) discriminator and an ESI for the particular Ethernet Segment.

This application claims priority to India Patent Application No.5257/CHE/2015, filed Oct. 1, 2015, the entire content of which isincorporated herein by reference

TECHNICAL FIELD

The invention relates to computer networks and, more specifically, toforwarding network traffic within computer networks.

BACKGROUND

A computer network is a collection of interconnected computing devicesthat can exchange data and share resources. Example network devicesinclude layer two devices that operate within the second layer (L2) ofthe Open Systems Interconnection (OSI) reference model, i.e., the datalink layer, and layer three devices that operate within the third layer(L3) of the OSI reference model, i.e., the network layer. Networkdevices within computer networks often include a control unit thatprovides control plane functionality for the network device andforwarding components for routing or switching data units.

An Ethernet Virtual Private Network (EVPN) may be used to extend two ormore remote layer two (L2) customer networks through an intermediatelayer three (L3) network (usually referred to as a provider network), ina transparent manner, i.e., as if the intermediate L3 network does notexist. In particular, the EVPN transports L2 communications, such asEthernet packets or “frames,” between customer networks via theintermediate network. In a typical configuration, provider edge (PE)network devices (e.g., routers and/or switches) coupled to the customeredge (CE) network devices of the customer networks define label switchedpaths (LSPs) (also referred to as pseudowires) within the providernetwork to carry encapsulated L2 communications as if these customernetworks were directly attached to the same local area network (LAN). Insome configurations, the PE network devices may also be connected by anIP infrastructure in which case IP/GRE tunneling or other IP tunnelingcan be used between the network devices.

In an EVPN, MAC learning between PE network devices occurs in thecontrol plane rather than in the data plane (as happens with traditionalbridging) using a routing protocol. For example, in EVPNs, a PE networkdevice typically uses the Border Gateway Protocol (BGP) (i.e., an L3routing protocol) to advertise to other provider edge network devicesthe MAC addresses learned from the local consumer edge network devicesto which the PE network device is connected. A PE device may use BGProute advertisement message to announce reachability information for theEVPN, where the BGP route advertisement specifies one or more MACaddresses learned by the PE network device instead of L3 routinginformation.

In an EVPN configuration referred to as all-active mode, an Ethernetsegment includes multiple PE network devices that provide multi-homedconnectivity for one or more local customer network devices. Moreover,the multiple PE network device provide transport services through theintermediate network to a remote PE network device, and each of themultiple PE network devices in the Ethernet segment operates forwardsEthernet frames in the segment for the customer network device. When anetwork failure occurs at one of the multiple PE network devices thatprovide multi-homed connectivity, an upstream PE network device in theEVPN may detect the failure via BGP and withdraw the failed network PEdevice. However, the withdrawal period may take multiple seconds, duringwhich network traffic is still forwarded to the failed network PE deviceand dropped.

SUMMARY

The techniques described herein enable a remote PE router to improveconvergence time in response to a link failure in an EVPN byestablishing per-Ethernet Segment Identifier (ESI) BidirectionalForwarding Detection (BFD) sessions with other PE routers that arecoupled to the PE router in EVPN active-active mode. For instance, inresponse to receiving Ethernet Auto-Discovery routes from the other PErouters, the remote PE router may determine that each of the other PErouters are included in the same Ethernet Segment. The remote PE routermay send ESI Ping request packets, which include the ESI for theEthernet Segment and respective BFD discriminators generated by theremote PE router, to each of the other PE routers. In response toreceiving ESI Ping reply packets from the other PE routers, the remotePE router may execute per-ESI BFD sessions with each of the other PErouters. Using EVPN label stacks, BFD packets for the per-ESI BFDsessions are tunneled through the EVPN between the remote PE router andthe respective other PE routers.

If a communication link in the Ethernet Segment fails between a customeredge router and a particular PE router that is executing a per-ESI BFDsession, the particular PE router terminates or otherwise interrupts theper-ESI BFD session with the remote PE router for the Ethernet Segment.In response to determining that the per-ESI BFD session has beeninterrupted or terminated, the remote PE router may update itsforwarding information to re-direct network traffic to other PE routersin the Ethernet segment. In this way, the remote PE router can bypassthe particular PE router that initially terminated the per-ESI BFDsession because traffic can longer flow in the Ethernet Segment from theparticular PE router to the customer edge router. By immediatelyre-directing traffic network upon failure of the per-ESI BFD session,techniques of the disclosure may prevent fewer dropped packets thanwaiting for a control plane message to withdraw the particular PE routerfrom the Ethernet Segment.

In some examples, a method includes determining, by a remote provideredge (PE) router coupled by an intermediate network to at least two PErouters operating in an Ethernet Virtual Private Network (EVPN) with theremote PE router, that the at least two PE routers are locally connectedto a multi-homed customer network by a particular Ethernet Segment; andsending, by the remote PE router and based on determining that the atleast two PE routers operating in the EVPN are connected to themulti-homed customer network by the particular Ethernet Segment, anEthernet Segment Identifier (ESI) Ping request packet through theintermediate network to one of the at least two PE routers, wherein theESI Ping request packet includes at least a Bidirectional ForwardingDetection (BFD) discriminator and an ESI for the particular EthernetSegment that locally connects the at least two PE routers to themulti-homed customer network.

In some examples, a remote PE router is coupled by an intermediatenetwork to at least two PE routers operating in an Ethernet VirtualPrivate Network (EVPN) with the remote PE router, wherein the remote PErouter includes: a routing engine that determines that the at least twoPE routers are locally connected to a multi-homed customer network by aparticular Ethernet Segment; and a forwarding unit that sends, based onthe routing engine determining that the at least two PE routersoperating in the EVPN are connected to the multi-homed customer networkby the particular Ethernet Segment, an Ethernet Segment Identifier (ESI)Ping request packet through the intermediate network to one of the atleast two PE routers, wherein the ESI Ping request packet includes atleast a Bidirectional Forwarding Detection (BFD) discriminator and anESI for the particular Ethernet Segment that locally connects the atleast two PE routers to the multi-homed customer network.

In some examples, a computer-readable medium includes instructions forcausing at least one programmable processor of a remote provider edge(PE) router, to: determine, by the remote provider edge (PE) routercoupled by an intermediate network to at least two PE routers operatingin an Ethernet Virtual Private Network (EVPN) with the remote PE router,that the at least two PE routers are locally connected to a multi-homedcustomer network by a particular Ethernet Segment; and send, based ondetermining that the at least two PE routers operating in the EVPN areconnected to the multi-homed customer network by the particular EthernetSegment, an Ethernet Segment Identifier (ESI) Ping request packetthrough the intermediate network to one of the at least two PE routers,wherein the ESI Ping request packet includes at least a BidirectionalForwarding Detection (BFD) discriminator and an ESI for the particularEthernet Segment that locally connects the at least two PE routers tothe multi-homed customer network.

The details of one or more embodiments of the disclosure are set forthin the accompanying drawings and the description below. Other features,objects, and advantages of the disclosure will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example system, in accordancewith techniques of the disclosure.

FIG. 2 is a block diagram illustrating an exemplary PE router capable ofperforming the disclosed techniques.

FIG. 3 is a flowchart illustrating example operations of multiplenetwork devices in accordance with techniques of the disclosure.

FIGS. 4A-4B are block diagrams illustrating an exemplary format of anESI Ping request packet and EVPN ESI FEC information, in accordance withtechniques of the disclosure.

FIG. 5 is a block diagram illustrating an exemplary format of aBidirectional Forwarding Detection packet for a per-ESI BFD session, inaccordance with techniques of the disclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example system, in accordancewith techniques of the disclosure. In the example of FIG. 1, PE routers10A-10C (“PE routers 10”) provide customer devices 4A-4D (“customerdevices 4”) associated with customer networks 6A-6B (“customer networks6”) with access to service provider network 12 via CE routers 8A-8B (“CErouters 8”). Communication links 16A-16G may be Ethernet, ATM or anyother suitable network connections.

PE routers 10 and CE routers 8 are illustrated as routers in the exampleof FIG. 1. However, techniques of the disclosure may be implementedusing switches or other suitable network devices that participate in alayer two (L2) virtual private network service, such as an EthernetVirtual Private Network (EVPN). Customer networks 6 may be networks forgeographically separated sites of an enterprise. Each of customernetworks 6 may include additional customer equipment 4A-4D (“customerequipment 4”), such as, one or more non-edge switches, routers, hubs,gateways, security devices such as firewalls, intrusion detection,and/or intrusion prevention devices, servers, computer terminals,laptops, printers, databases, wireless mobile devices such as cellularphones or personal digital assistants, wireless access points, bridges,cable modems, application accelerators, or other network devices. Theconfiguration of network 2 illustrated in FIG. 1 is merely exemplary.For example, an enterprise may include any number of customer networks6. Nonetheless, for ease of description, only customer networks 6A-6Bare illustrated in FIG. 1.

Service provider network 12 represents a publicly accessible computernetwork that is owned and operated by a service provider, which isusually large telecommunications entity or corporation. Service providernetwork 12 is usually a large layer three (L3) computer network, wherereference to a layer followed by a number refers to a correspondinglayer in the Open Systems Interconnection (OSI) model. Service providernetwork 12 is a L3 network in the sense that it natively supports L3operations as described in the OSI model. Common L3 operations includethose performed in accordance with L3 protocols, such as the Internetprotocol (IP). L3 is also known as a “network layer” in the OSI modeland the term L3 may be used interchangeably with the phrase “networklayer” throughout this disclosure.

Although not illustrated, service provider network 12 may be coupled toone or more networks administered by other providers, and may thus formpart of a large-scale public network infrastructure, e.g., the Internet.Consequently, customer networks 6 may be viewed as edge networks of theInternet. Service provider network 12 may provide computing deviceswithin customer networks 6 with access to the Internet, and may allowthe computing devices within the customer networks to communicate witheach other.

Service provider network 12 may include a variety of network devicesother than PE routers 10. For instance, service provider network 12 mayinclude a route reflector 18. In some examples, route reflector 18 mayreside within service provider network 12 and along a path in serviceprovider network 12 between two or more PE routers. Because of aninternal BGP (IBGP) full-mesh requirement, some networks may use routereflectors to simplify configuration. Using a route reflector, routersare grouped into clusters, which are identified by numeric identifiersunique to an autonomous system (AS). Within the cluster, a BGP sessionis configured from a single router (the route reflector) to eachinternal peer. With this configuration, the IBGP full-mesh requirementmay be met by route reflector 18. To use route reflection in an AS, oneor more routers are designated as a route reflector—typically, one perpoint of presence (POP). Route reflectors have the BGP ability tore-advertise routes learned from an internal peer to other internalpeers. Rather than requiring all internal peers to be fully meshed witheach other, route reflection may have only the route reflector be fullymeshed with all internal peers.

Although additional network devices are not shown for ease ofexplanation, it should be understood that system 2 may compriseadditional network and/or computing devices such as, for example, one ormore additional switches, routers, hubs, gateways, security devices suchas firewalls, intrusion detection, and/or intrusion prevention devices,servers, computer terminals, laptops, printers, databases, wirelessmobile devices such as cellular phones or personal digital assistants,wireless access points, bridges, cable modems, application accelerators,or other network devices. Moreover, although the elements of system 2are illustrated as being directly coupled, it should be understood thatone or more additional network elements may be included along any ofnetwork links 16, such that the network elements of system 2 are notdirectly coupled.

Service provider network 12 typically provides a number of residentialand business services, including residential and business class dataservices (which are often referred to as “Internet services” in thatthese data services permit access to the collection of publicallyaccessible networks referred to as the Internet), residential andbusiness class telephone and/or voice services, and residential andbusiness class television services. One such business class data serviceoffered by service provider network 12 includes L2 EVPN service. Forexample, an EVPN is a service that provides a form of L2 connectivityacross an intermediate L3 network, such as service provider network 12,to interconnect two L2 customer networks, such as L2 customer networks6, that are usually located in two different geographic areas. Often,EVPN is transparent to the customer networks in that these customernetworks are not aware of the intervening intermediate service providernetwork and instead act and operate as if these two customer networkswere directly connected and formed a single L2 network. In a way, EVPNenables a form of a transparent LAN connection between twogeographically distant customer sites that each operates a L2 networkand, for this reason, EVPN may also be referred to as a “transparent LANservice.”

To configure an EVPN, a network operator of the service providerconfigures various devices included within service provider network 12that interface with L2 customer networks 6. The EVPN configuration mayinclude an EVPN instance (EVI), which consists of one or more broadcastdomains. Generally, an EVI may refer to a routing and forwardinginstance on a PE router, such as PE routers 10A-10C. Consequently,multiple EVIs may be configured on PE routers 10 for Ethernet segment14, as further described herein, each providing a separate, logicallayer two (L2) forwarding domain. In this way, multiple EVIs may beconfigured that each includes one or more of PE routers 10A-10C ofEthernet segment 14. In some examples, Ethernet Tags are then used toidentify a particular broadcast domain, e.g., a VLAN, in an EVI. A PErouter may advertise a unique EVPN label per <ESI, Ethernet Tag>combination. This label assignment methodology is referred to as a per<ESI, Ethernet Tag> label assignment. Alternatively, a PE router mayadvertise a unique EVPN label per MAC address. In still another example,a PE router may advertise the same single EVPN label for all MACaddresses in a given EVI. This label assignment methodology is referredto as a per EVI label assignment.

In the example of FIG. 1, for use in transporting communicationsassociated with one or more EVIs, the network operator configures PErouters 10 to provision pseudowires 17A-17C for transporting L2communications. Pseudowires are logical network connections formed fromtwo unidirectional label switched paths (LSPs) that emulate a connectionnot natively offered by service provider network 12 for consumptionoutside the boundaries of that service provider network 12. Pseudowiresmay emulate a L2 connection within service provider network 12 enablingservice provider network 12 to offer emulated L2 connectivity externallyfor consumption by L2 customer networks 6. As such, each EVPN instancemay operate over pseudowires 17 to enable a logical form of L2connectivity between customer networks 6.

To configure an EVI, pseudowires 17 may be configured such that each ofPE routers 10 that provide a given EVI is interconnected by way ofpseudowires to every other one of the PE devices participating in theEVI. In the example of FIG. 1, each of PE routers 10 provides access tothe EVPN for carrying traffic associated with customer networks 6 and,therefore, each of PE devices 10 within the same Ethernet segment may beconnected to every other PE device 10 via pseudowires 17. Oncepseudowires are configured in this manner, EVPN may be enabled within PEdevices 10 to operate over the pseudowires, which may in this contextoperate as logical dedicated links through service provider network 12.In operation, EVPN generally involves prepending or otherwise insertinga tag and a pseudowire label onto incoming L2 packets, which may also bereferred to as L2 frames (particularly in the context of Ethernet), andtransmitting the tagged packets through a corresponding one of theconfigured pseudowires. Once EVPN is configured within service providernetwork 12, customer devices 4 within customer networks 6 maycommunicate with one another via EVPN as if they were directly connectedL2 networks.

In order to establish the EVPN, an EVPN protocol executing on PE routers10A-10C triggers EVPN designated forwarder (DF) election for Ethernetsegment 14. This may be accomplished, for example, by EVPN protocolexecuting on each of PE routers 10A-10C that participates in theEthernet segment directing the router to output a routing protocolmessage advertising an Ethernet Segment Identifier (ESI), which istypically unique across all EVPN instances (EVIs). In addition, for eachEVI, the EVPN protocol directs the router to output a routing protocolmessage advertising an Ethernet Auto-Discovery (AD) route specifying therelevant ESI for the Ethernet segment coupled to the EVPN instance. Oncethe EVPN is operational for the {EVI, ESI} pair, PE routers 10A-10Coutput routing protocol messages to remote PE router 10D to announcemedia access control (MAC) addresses associated with customer equipmentin customer network 6B.

For example, in typical operation, PE routers 10A-10C communicate usingthe Border Gateway Protocol (BGP) and the EVPN protocol specifies BGPNetwork Layer Reachability Information (NLRI) for the EVPN and maydefine different route types for conveying EVPN information via the BGProuting protocol. The EVPN NLRI is typically carried in BGP using BGPMultiprotocol Extensions. An Ethernet Segment route advertised by eachPE router 10A-10C using BGP includes a Route Distinguisher and EthernetSegment Identifier. An Ethernet AD route advertised by each PE router10A-10C for each EVI, specifies a Route Distinguisher (RD) (e.g., an IPaddress of an MPLS Edge Switch (MES)), ESI, Ethernet Tag Identifier, andMPLS label. Subsequent BGP media access control (MAC) routes output byPE router 10A-10C announce MAC addresses of customer equipment 4 for theEVPN include a RD, ESI, Ethernet Tag Identifier, MAC address and MACaddress length, IP address and IP address length, and MPLS label.

In active-standby mode in which one of PE router 10A or 10B forwardsnetwork traffic from PE router 10C to CE router 8B, the EVPN protocolexecuting on each PE router 10A-10C initiates EVPN DF election for theEthernet segment on a per-EVPN instance basis, and participates withinthat election for each EVPN instance. That is, DF election may be at thegranularity of each ESI, EVI combination. If elected DF, one of PErouters 10A-10C elected as DF forwards traffic from the EVPN to local CErouter 8B. Additional example information with respect to the EVPNprotocol is described in “BGP MPLS Based Ethernet VPN,”draft-ietf-l2vpn-evpn-11, Internet Engineering Task Force (IETF), Jul.2, 2014, the entire contents of which are incorporated herein byreference.

In the example of FIG. 1, when providing the EVPN service to customernetworks 6, PE routers 10 and CE routers 8 typically perform MAC addresslearning to efficiently forward L2 network communications in system 2.That is, as PE routers 10 and CE routers 8 forward Ethernet frames, therouters learn L2 state information for the L2 network, including MACaddressing information for customer equipment 4 within the network andthe physical ports through which customer equipment 4 are reachable. PErouters 10 and CE routers 8 typically store the MAC addressinginformation in MAC tables associated with respective interfaces. Whenforwarding an individual Ethernet frame received on one interface, arouter typically broadcasts the Ethernet frame to all other interfacesassociated with the EVPN unless the router has previously learned thespecific interface through which the destination MAC address specifiedin the Ethernet frame is reachable. In this case, the router forwards asingle copy of the Ethernet frame out the associated interface.

Moreover, as PE routers 10 learn the MAC address for customer equipment4 reachable through local attachment circuits, the PE routers 10 utilizeMAC address route advertisements of a layer three (L3) routing protocol(i.e., BGP in this example) to share the learned MAC addresses and toprovide an indication that the MAC addresses are reachable through theparticular PE router that is issuing the route advertisement. In theEVPN implemented using PE routers 10 for a given EVI, each of PE routers10 advertises the locally learned MAC addresses to other PE routers 10using a BGP route advertisement, also referred to herein as a “MACroute” or a “MAC Advertisement route.” As further described below, a MACroute typically specifies an individual MAC address of customerequipment 4 along with additional forwarding information, such as aroute descriptor, route target, layer 2 segment identifier, MPLS label,etc. In this way, PE routers 10 use BGP to advertise and share the MACaddresses learned when forwarding layer two communications associatedwith the EVPN. Accordingly, PE routers 10 may perform both locallearning and remote learning of MAC addresses.

Each of PE routers 10 (e.g., PE router 10D) utilizes MAC routesspecifying the MAC addresses learned by other PE routers to determinehow to forward L2 communications to MAC addresses that belong customerequipment 4 connected to other PEs, i.e., to remote CE routers and/orcustomer equipment behind CE routers operatively coupled to PE routers.That is, each of PE routers 10 determine whether Ethernet frames can besent directly to a particular one of the other PE routers 10 or whetherto treat the Ethernet frames as so called “BUM” traffic (Broadcast,Unidentified Unicast or Multicast traffic) that is to be flooded withinthe EVPN based on the MAC addresses learning information received fromthe other PE routers.

As shown in FIG. 1, CE routers 8 may be multi- and/or singly-homed toone or more of PE routers 10. In EVPN, a CE router may be said to bemulti-homed when it is coupled to two physically different PE routers onthe same EVI when the PE routers are resident on the same physicalEthernet Segment. As one example, CE router 8B is coupled to PE routers10A and 10B via links 16D and 16E, respectively, where PE routers 10Aand 10B are capable of providing access to EVPN for L2 customer network6B via CE router 8B. In instances where a given customer network (suchas customer network 6B) may couple to service provider network 12 viatwo different and, to a certain extent, redundant links, the customernetwork may be referred to as being “multi-homed.” In this example, CErouter 8B may be multi-homed to PE routers 10A and 10B because CE router8B is coupled to two different PE routers 10A and 10B via separate and,to a certain extent, redundant links 16D and 16E where both of PErouters 10A and 10B are capable of providing access to EVPN for L2customer network 6B. Multi-homed networks are often employed by networkoperators so as to improve access to EVPN provided by service providernetwork 12 should a failure in one of links 16D, 16E, and 16F occur. Ina typical EVPN configuration, only the multi-homing PEs 10A-10Bparticipate in DF election for each ESI. PE 10C not connected to the ESIhas no direct knowledge of the DF election result for a give ESI.

In active-active mode configurations, remote PE 10C is typicallyconfigured to install the MAC routes for the ESI, such that trafficdownstream traffic to customer network 6B is balanced between PE routers10A and 10B, which are each included in the ESI. In the example of FIG.1, PE routers 10A and 10B may be configured in EVPN active-active mode,such that PE router 10C load-balances downstream network traffic tocustomer network 6B between PE routers 10A and 10B. In active-activemode, each of PE routers 10A and 10B may be configured as part of thesame Ethernet Segment and therefore have the same Ethernet SegmentIdentifier. PE routers 10A and 10B may each advertise Ethernet AD routesto PE router 10C that specify respective IP addresses and ESIs for eachof PE routers 10A and 10B. In this way, PE router 10C may configure oneor more of its forwarding units (or “forwarding engines”) to loadbalance network traffic destined for customer network 6B between PErouters 10A and 10B.

An EVPN, such as illustrated in FIG. 1, may operate over anMulti-Protocol Label Switching (MPLS) configured network and use MPLSlabels to forward network traffic accordingly. MPLS is a mechanism usedto engineer traffic patterns within Internet Protocol (IP) networksaccording to the routing information maintained by the routers in thenetworks. By utilizing MPLS protocols, such as the Label Distributionprotocol (LDP) or the Resource Reservation Protocol with TrafficEngineering extensions (RSVP-TE), a source device can request a paththrough a network to a destination device, i.e., a Label Switched Path(LSP). An LSP defines a distinct path through the network to carry MPLSpackets from the source device to a destination device. Using a MPLSprotocol, each router along an LSP allocates a label and propagates thelabel to the closest upstream router along the path. Routers along thepath add or remote the labels and perform other MPLS operations toforward the MPLS packets along the established path.

As shown in the example of FIG. 1, PE routers 10A-10C may provide anMPLS core for sending network packets from customer network 6A to andfrom customer network 6B. Each of PE routers 10A-10C implement the MPLSprotocol and apply one or more MPLS labels, i.e., a label stack, tonetwork packets in accordance with routing and forwarding informationconfigured at each respective PE router. In an EVPN, a label stackapplied to a network packet may include multiple labels. For instance, alabel stack may include an outer label and an inner label.

The outer label serves as a “transport label” that uniquely identifies aPE router in an MPLS core. That is, each of PE routers 10A-10C mayexchange control plane messages at configuration and startup thatspecify an outer label that uniquely identifies each respective PErouter. For instance, PE router 10A may send control plane messages thatspecify an outer label that identifies PE router 10A to PE routers10B-10C. PE routers 10B-10C may configure their respective forwardingunits such that network packets that include the outer labelcorresponding to PE router 10A are forwarded to PE router 10A.

The inner label, or “service label,” of the MPLS label stack providesEVPN-specific configuration information. As described above, EVPNdefines Ethernet AD routes, MAC advertisement routes, and EthernetSegment routes. An Ethernet AD route, for example, may be structuredaccording to the following format of Table 1:

TABLE 1 AD route advertisement Route Descriptor (8 octets) EthernetSegment Identifier (10 octets) Ethernet Tag ID (4 octets) MPLS AliasLabel (3 octets)In one example, PE router 10A may send an Ethernet AD route to PE router10C initially at startup and configuration that includes an MPLS labelas shown above. PE router 10C may configure one or more of itsforwarding units to apply the MPLS label of the Ethernet AD route fromPE router 10A as the inner label in a label stack applied to networkpackets that are destined to PE router 10A. PE router 10C would thenapply the transport label identifying PE router 10A as the outer labelin the label stack. In this way, the inner label providesEVPN-specification configuration information about the Ethernet AD routethat PE router 10C uses to forward network packets through the EVPN.

In EVPN architectures, such as FIG. 1, a link may fail, such as 16Bbetween PE router 10B and CE router 8B. In such examples, PE router 10Cmay determine that link 17B has failed upon PE router 18B notifying PErouter 10C via BGP that network traffic cannot reach CE router 8B fromPE router 10B. Once PE router 10C has determined that PE router 10B isno longer available to forward network traffic to customer network 6B,PE router 10C may take corrective action by removing PE router 10B froman adjacency list that includes the members of the Ethernet Segment thatpreviously included PE routers 10A and 10B. PE router 10C, upon updatingthe adjacency list, may then start sending traffic only to PE router 10Aand stop load balancing network traffic between PE routers 10A and 10B.

Because BGP is a control-plane messaging protocol, PE router 10C may notdetermine for a number of seconds that link 16E has failed or thatnetwork traffic is no longer flowing from PE router 10C to CE router 8Bvia PE router 10B. Until PE router 10A has taken the corrective actionto remove PE router 10B from the adjacency list and start sendingtraffic only to PE router 10A, PE router 10C may continue sendingtraffic for, potentially a number seconds, to PE router 10B although thetraffic will be dropped at PE router 18B because communication link 16Bhas failed.

Techniques of this disclosure may reduce the amount of time that networktraffic is dropped at PE router 18B in the event that network trafficcannot reach CE router 8B from PE router 10C via PE router 10B. Byre-directing network traffic away from PE router 10B in the event of afailure of link 16E or at a network device on a path between routereflector 18 and PE router 10B (e.g., a network device that forms partof pseudowire 17B), the amount of time that network traffic is droppedmay be reduced from seconds to an order of milliseconds, such as lessthan 100 or 50 milliseconds. Techniques of the disclosure may configurePE router 10C to run Bidirectional Forwarding Detection (BFD) over EVPNon a per-ESI basis. That is, PE router 10C may run BFD sessions on aper-ESI basis with each PE router operating in active-active mode in thesame Ethernet Segment (e.g., having the same ESI). In this way, if PErouter 10C determines for example, via a per-ESI BFD session with PErouter 10B, that network traffic cannot reach CE router 8B from via PErouter 10B, then PE router 10C may immediately re-direct network trafficfor the Ethernet Segment away from PE router 10B and to other PE routersthat are coupled to PE router 10C in active-active mode in the sameEthernet Segment, such as PE router 10A.

In the example of FIG. 1, each of PE routers 10A-10C runs EVPN and BFDprotocols. At initial configuration and startup, each of PE routers 10Aand 10B may advertise Ethernet AD routes, as described above. PE router10C inspect the contents of each Ethernet AD route and determine that anESI for an Ethernet AD route advertised by PE router 10B matches an ESIfor an Ethernet AD route advertised by PE router 10A. PE router 10C mayalso determine that PE router 10C is coupled to each of PE routers 10Aand 10B, which are configured in EVPN active-active mode for EthernetSegment 14 having an ESI 200. Based on determining that PE router 10C iscoupled to each of PE routers 10A and 10B operating in active-activemode for Ethernet Segment 14, PE router 10C may send Ethernet SegmentIdentifier Ping (“ESI Ping”) request packets to each of PE routers 10Aand 10B. More generally, PE router 10C may send ESI Ping request packetsto exercise all paths of an ESI.

As further described in this disclosure, an ESI Ping request packet maybe used to initiate and establish a BFD session on a per-ESI basisbetween two PE routers in an EVPN. In some examples, an ESI Ping requestpacket may be a Label-Switch Path Ping (“LSP Ping”) exploration messagethat further includes a label stack for routing the packet within anEVPN. Additional example information with respect to an LSP Ping messageis described in U.S. Pat. No. 7,852,778 “Verification of Network PathsUsing Two or More Connectivity Protocols,” which issued on Dec. 14,2010, the entire contents of which are incorporated herein by reference.Additional example information with respect to an LSP Ping message isalso described in RFC 5884 “Bidirectional Forwarding Detection (BFD) forMPLS Label Switched Paths (LSPs),” June 2010, the entire contents ofwhich are incorporated herein by reference. The label stack for routingthe ESI Ping request packet within an EVPN may include an outertransport label and an inner service label as described above.

As an example, upon determining that PE router 10C is coupled to each ofPE routers 10A and 10B operating in active-active mode for EthernetSegment 14, PE router 10C may generate ESI Ping requests that PE router10C sends to PE routers 10A and 10B. For instance, PE router 10C maygenerate the ESI Ping request packet as an LSP ping exploration packetfor PE router 10B with an attached EVPN label stack that includes anouter transport label for a next hop on a path to PE router 10B andinner service label that represents the MPLS Alias Label, which PErouter 10B previously advertised to PE router 10C in an Ethernet ADroute. Additional example information with respect to an LSP pingexploration packet is also described in RFC 5884.

The ESI Ping request packet may include, but is not limited to: a localdiscriminator (or “My Discriminator”) generated by PE router 10C basedon the LSP-Ping protocol that uses the BFD protocol. Further details ofthe BFD protocol can be found in RFC 5880 “Bidirectional ForwardingDetection (BFD),” June 2010, the entire contents of which areincorporated herein by reference. The local discriminator may correspondto or identify a participant in a particular BFD session, such as PErouter 10C. As later described in this disclosure, PE router 10B maysend an ESI Ping reply to PE router 10C that includes the localdiscriminator of PE router 10C. In this way, PE router 10C may determinethat PE router 10B is a participant in the initiated BFD session and hassuccessfully established a BFD session with PE router 10C. In someexamples, the ESI Ping reply from PE router 10B to PE router 10C may bea BFD Control message that further includes a label stack with an outertransport label for a next hop in a path from PE router 10B to PE router10C, and inner service label that represents the MPLS Alias Label, whichPE router 10C previously advertised to PE router 10B in an Ethernet ADroute.

PE router 10B may receive the ESI Ping request packet that is sent by PErouter 10C. In some examples, PE router 10C may set the Time-to-Live(TTL) value in the ESI Ping request packet to 1, such that PE router 10Bwill not forward the ESI Ping request packet into ESI 200, but ratherprocess the ESI Ping request packet in the routing engine (or controlplane) of PE router 10B. Upon receiving the ESI Ping request, PE router10B may determine that the received message is an ESI Ping requestpacket based on its contents. For instance PE router 10B may run BFD andEVPN protocols. Based on determining that the message includes an EVPNlabel stack of inner service label and outer transport label, andfurther that the contents of the message includes a BFD localdiscriminator, PE router 10B may generate an ESI Ping reply message. TheESI Ping reply message may be a LSP ping exploration-reply that furtherincludes a label stack with an outer transport label for a next hop in apath from PE router 10B to PE router 10C, and inner service label thatrepresents the MPLS Alias Label, which PE router 10C previouslyadvertised to PE router 10B in an Ethernet AD route. The messagegenerated by PE router 10B in response to the ESI Ping request packetmay include the BFD local discriminator received from PE router 10B anda BFD remote discriminator (or “Your Discriminator”) that corresponds toor identifies PE router 10B as a participant in the particular BFDsession. PE router 10B may send the ESI Ping reply back to PE router10C.

PE router 10C, upon receiving the ESI Ping reply message from PE router10C, may determine that the ESI Ping reply message includes the localdiscriminator of PE router 10B and a remote discriminator of PE router10C for a BFD session. PE router 10C may utilize the BFD protocol as alight-weight means of testing the data plane between PE router 10C andPE router 10B. The light-weight nature of the BFD protocol may allow PErouter 10C to implement the BFD protocol in hardware or firmware. TheBFD protocol may provide faster detection of data plane failures withsub-second granularity. For example, PE router 10C may execute alow-cost BFD test of a path between PE router 10C and PE router 10B forESI 200 at a higher frequency, e.g., once every 10 milliseconds, thanbetween iterations of the more extensive EVPN protocol that is used toperiodically verify the control plane against the data plane between PErouters 10C and 10B.

As shown in FIG. 1, upon receiving ESI Ping reply message from PE router10B, PE route 10C may establish and execute a BFD session 20B between PErouter 10C and PE router 10B. Additionally, upon receiving ESI Pingreply message from PE router 10A, PE route 10C may establish and executea BFD session 20A between PE router 10C and PE router 10A. Although thefollowing example of FIG. 1 is described with respect to BFD session20B, similar techniques may be applied with respect to BFD session 20A.

PE router 10C and PE router 10B may exchange BFD packets, which includethe local and remote discriminators, in the BFD session according to amode specified in RFC 5880, such as Asynchronous or Demand mode, orusing an adjunct Echo function. The BFD packets may include an EVPNlabel stack having an outer transport label and inner service label toroute the packets between PE routers for a specific Ethernet Segment,such as ESI 200. In this way, a BFD session may be established on aper-ESI basis because the BFD packets have EVPN label stack thatincludes an inner service label that corresponds to a particular ESI foran Ethernet Segment. As such, PE router 10C may determine that a BFDsession has terminated or has been interrupted if either communicationlink 16E fails or network device or link on a path between routereflector 18 and PE router 10B fails. That is, if either communicationlink 16E fails or a network device or a link on a path between routereflector 18 and PE router 10B fails, PE router 10C may no longerreceive BFD reply messages that correspond to a BFD session for ESI 200from PE router 10B. As an example, if communication link 16E fails, PErouter 10B may not send BFD reply messages for BFD session 20B thatcorresponds to ESI 200 because PE router 10B cannot forward networkpackets to CE router 8B for ESI 200.

PE router 10C may install respective forwarding next hops for each of PErouters 10A and 10B in forwarding structures (e.g., forwarding tables,context tables, radix trees, next hop lists, chained next hops, or thelike) of one or more of PE router 10C's forwarding units to forwardnetwork packets. A forwarding next hop for PE router 10A may specify oneor more operations that cause PE router 10C to forward a network packetout an egress interface that couples PE router 10C to PE router 10A.Similarly, a forwarding next hop for PE router 10B may cause PE router10C to forward a network packet out an egress interface that couples PErouter 10C to PE router 10B.

PE router 10C may use the forwarding next hops to load balance trafficfrom PE router 10C between PE routers in Ethernet Segment 14, such as PErouter 10A and PE router 10B. When forwarding traffic, PE router 10C mayperform a lookup on packet header information of a packet and determinethat the packet is destined for a device in customer network 6B. Forexample, a lookup on a packet header may resolve to an entry in aforwarding structure such as a radix tree that corresponds to one ormore forwarding next hops. PE router 10C may use a hashing algorithm tovary the selection of the forwarding next hops for PE routers 10A and10B in order to load-balance network packets between PE routers 10A and10B, which are each included in Ethernet Segment 14. PE router 10C mayinitially load-balance the forwarding of network packets to PE routers10A and 10B while no failure has occurred at communication link 16E andno failure has occurred at a network device or a link on a path betweenroute reflector 18 and PE router 10B.

At a later time, communication link 16E may fail in the example ofFIG. 1. PE router 10B stop sending BFD reply messages for BFD session20B that corresponds to ESI 200 because PE router 10B cannot forwardnetwork packets to CE router 8B for ESI 200. PE router 10C may determinethat BFD session 20B has terminated or been interrupted and thereforenetwork traffic can no longer flow from PE router 10C to CE router 8Bvia a path that includes PE router 10B. As such, PE router 10C mayre-direct network traffic for Ethernet Segment 14 to other PE routersthat are also operating in active-active mode for Ethernet Segment 14,such as PE router 10A. To re-direct network traffic, PE router 10C mayinvalidate the forwarding next hop for PE router 10B. Accordingly, if PErouter 10C performs a lookup on a packet destined for a device incustomer network 6B, PE router 10C may forward the network packet usingthe operations of the forwarding next hop for PE router 10A. That is,invalidating the forwarding next hop for PE router 10B may remove aroute for PE router 10B from a forwarding structure of PE router 10C,such that traffic is no longer forwarded by PE router 10C to PE router10B. In this way, by detecting that BFD session 20B between PE router10C and PE router 10B for ESI 200 has been terminated or interrupted, PErouter 10C may stop load-balancing network traffic to PE router 10B,thereby reducing the number of packets dropped at PE router 10B orbetween route reflector 18 and PE router 10B after a failure of link16E. By using BFD sessions on a per-ESI basis between PE routers 10A and10B, PE router 10C may detect a link failure more quickly in the dataplane, than if PE router 10C eventually received a BGP message in thecontrol plane that indicated a route no longer existed between PE router10B and CE router 8B.

In the example of FIG. 1, after invalidating the forwarding next hop forPE router 10B, PE router 10C may later receive from PE router 10B, arequest to withdraw the Ethernet Auto-Discovery route for PE router 10B.That is, PE router 10B may send a BGP control plane message to indicatewithdrawal from Ethernet Segment 14 that corresponds to ESI 200. PErouter 10C, and more specifically routing engine 84 as further describedin FIG. 2, may update one or more forwarding units of PE router 10B toonly encapsulate network traffic with an EVPN label stack for PE router10A (e.g., the remaining PE routers after PE router 10B is withdrawn).PE router 10C, upon receiving the BGP control plane message to withdrawPE router 10B, may perform global repair by removing PE router 10B froman adjacency list maintained by PE router 10C that includes each memberof Ethernet Segment 14. The BGP control plane message to withdraw PErouter 10B from Ethernet Segment 14 may take seconds to send, receiveand process, due to route reflector 18 processing the control planemessage from PE router 10B and forwarding it to PE router 10C, andfurther due to processing time at PE router 10C to withdraw PE router10B. Techniques of the disclosure may use the per-ESI BFD sessions todetect failures more quickly and thereby re-direct network traffic forthe Ethernet Segment more quickly to reduce the number of packetsdropped at PE router 10B as a result of failure at link 16E.

FIG. 2 is a block diagram illustrating an exemplary PE router 10Ccapable of performing the disclosed techniques. In general, PE router10C may operate substantially similar to PE router 10C of FIG. 1. Inthis example, PE router 10C includes interface cards 88A-88N (“IFCs 88”)that receive packets via incoming links 90A-90N (“incoming links 90”)and send packets via outbound links 92A-92N (“outbound links 92”). IFCs88 are typically coupled to links 90, 92 via a number of interfaceports. PE router 10C also includes a control unit 82 that determinesroutes of received packets and forwards the packets accordingly via IFCs88.

Control unit 82 may comprise a routing engine 84 and a packet forwardingengine 86. Routing engine 84 operates as the control plane for PE router10C and includes an operating system that provides a multi-taskingoperating environment for execution of a number of concurrent processes.Routing engine 84, for example, execute software instructions toimplement one or more control plane networking protocols 97. Forexample, protocols 97 may include one or more routing protocols, such asBorder Gateway Protocol (BGP) 99 for exchanging routing information withother routing devices and for updating routing information 94. Protocols97 may also include Multiprotocol Label Switching Protocol (MPLS) 95 fortunneling packets within service provider network 12. Protocols 97 mayalso include Bidirectional Forwarding Detection (BFD) protocol 103.Protocols 97 may include LSP-Ping 105, which implements the LSP-pingprotocol of RFC 5884.

Routing protocol daemon (RPD) 99 may use protocols 97 to exchangerouting information, stored in routing information 94, with otherrouters. Routing information 94 may include information defining atopology of a network. RPD 99 may resolve the topology defined byrouting information in routing information 94 to select or determine oneor more routes through the network. RPD 99 may then generate forwardinginformation 106 and update forwarding plane 86 with routes fromforwarding information 106. Routing engine 84 executes EVPN protocol 87,which operates to communicate with other routers to establish andmaintain an EVPN, such as the EVPN of FIG. 1, for transporting L2communications through an intermediate network so as to logically extendan Ethernet network through the intermediate network. EVPN protocol 87may, for example, communicate with EVPN protocols executing on remoterouters.

Routing information 94 may describe a topology of the computer networkin which PE router 10C resides, and may also include routes through theshared trees in the computer network. Routing information 94 describesvarious routes within the computer network, and the appropriate nexthops for each route, i.e., the neighboring routing devices along each ofthe routes. Routing engine 84 analyzes stored routing information 94 andgenerates forwarding information 106 for forwarding engine 86.Forwarding information 106 may associate, for example, networkdestinations for certain multicast groups with specific next hops andcorresponding IFCs 88 and physical output ports for output links 92.Forwarding information 106 may be a radix tree programmed into dedicatedforwarding chips, a series of tables, a complex database, a link list, aradix tree, a database, a flat file, or various other data structures.

In some examples, forwarding engine 86 arranges forwarding structures asforwarding next hop data that can be chained together as a series of“forwarding next hops” along an internal packet forwarding path for thenetwork device. In many instances, the forwarding structures performlookup operations within internal memory of ASICs included in forwardingengine 86, where the lookup may be performed against a tree (or trie)search, a table (or index) search. Other example operations that may bespecified with the forwarding next hops include filter determination andapplication, or a rate limiter determination and application. Lookupoperations locate, within a lookup data structure (e.g., a lookup tree),an entry that matches packet contents or another property of the packetor packet flow, such as the inbound interface of the packet. The resultof packet processing in accordance with the operations defined by theforwarding next hop structure within ASICs determines the manner inwhich a packet is forwarded or otherwise processed by forwarding engine86 from its input interface on one of IFCs 88 to its output interface onone of IFCs 88.

In FIG. 2, forwarding engine 86 may receive Ethernet AD routes sent byPE routers 10A and 10B to PE router 10C. For instance, forwarding engine86 may receive a packet that specifies an Ethernet AD route from PErouter 10A and which includes the route descriptor, Ethernet SegmentIdentifier, Ethernet Tag ID and MPLS label. Forwarding engine 86 mayreceive an Ethernet AD route from PE router 10B that includes the routedescriptor, Ethernet Segment Identifier, Ethernet Tag ID and MPLS label.If forwarding engine 86 determines that the packet specifies an EthernetAD route, forwarding engine 86 may send the packet data to protectionmodule 101.

Protection module 101 may determine that the ESI for each of theEthernet AD routes is the same for at least two routers that are coupledPE router 10C and operating in EVPN active-active mode. As such, PErouter 10C may, in some examples, provide protection for one or more ofthe PE routers operating in active-active mode for the same EthernetSegment, such as PE router 10B. For example, protection module 101 mayinitially request an LSP-Ping formatted packet or information togenerate an LSP-Ping packet from LSP-Ping 105. In some examples,protection module 101 requests a local discriminator from BFD 103, whichprotection module 101 may include in the LSP-Ping packet. Protectionmodule 101 may further determine from EVPN 87 an inner service label (orMPLS Alias Label) that PE router 10B previously advertised for ESI 200that corresponds to Ethernet Segment 14. Protection module 101 may alsodetermine an outer transport label that corresponds to a next hop fromPE router 10C to PE router 10B. Protection module 101 may assemble theESI Ping request, as the LSP-Ping packet, which further includes thelocal discriminator received from BFD 103 and an attached EVPN labelstack that includes the outer transport label and inner service labeldetermined from EVPN 87. As described in FIG. 1, protection module 101may set the TTL value for the ESI Ping request packet to 1. Protectionmodule 101 may store state information that associates the ESI for theinner service label with the local discriminator received from BFD 103,such that protection module 101 can identify a BFD session 20B for thelocal discriminator on a per-ESI basis.

Protection module 101 may then send the packet to forwarding engine 86,which forwards the ESI Ping request packet to PE router 10B using IFC88B. As described in FIG. 1, PE router 10B receives the ESI Pingrequest. PE router 10B generates a remote discriminator and sends an ESIPing reply message to PE router 10C that includes, but is no limited to,the remote discriminator generated by PE router 10B and the localdiscriminator generator by PE router 10C. PE router 10C may receive theESI Ping reply message from PE router 10B via IFC 88B. Forwarding engine106 may determine that the ESI Ping reply message is in response to theESI Ping request packet and send to protection module 101 for furtherprocessing. For instance, forwarding engine 86 may determine that theESI Ping reply message includes the local discriminator previously sentin the ESI Ping request packet and further includes a remotediscriminator generated by PE router 10B.

Protection module 101 may determine that the local discriminator of theESI Ping reply message matches the local discriminator previouslygenerated by BFD 103 for BFD session 20B with PE router 10B for ESI 200of Ethernet Segment 14. Protection module 101 may store stateinformation that further associates the local discriminator receivedfrom PE router 10B with the local discriminator generated by PE router10C, such that protection module 101 can identify BFD session 20B forthe local and remote discriminators on a per-ESI basis. Protectionmodule 101 may send the local and remote discriminators to BFD 103 torequest that BFD 103 initiate and execute BFD session 20B with PE router10B using BFD module 107 that is implemented in forwarding engine 86.BFD 103 may send the local and remote discriminators to BFD module 107,which executes BFD session 20B with PE router 10B. To execute BFDsession 20B with PE router 10B, BFD module 107 may generate BFD packetsin accordance with RFC 5880. However, BFD module 107 may further attachan EVPN label stack including an outer transport label and inner servicelabel to the BFD packets in order to route the BFD packets to PE router10B. For instance, BFD module 107 may determine the outer transportlabel and inner service label from forwarding information 106, BFD 103and/or protection module 101. The inner service label (or MPLS AliasLabel) may have been previously advertised from PE router 10B for ESI200 that corresponds to Ethernet Segment 14, and the outer transportlabel corresponds to a next hop from PE router 10C to PE router 10B.

BFD module 107 may cause forwarding engine 86 to execute BFD session 20Bby exchanging BFD packets with PE router 10B, which include the localand remote discriminators, in BFD session 20B according to a modespecified in RFC 5880, such as Asynchronous or Demand mode, or using anadjunct Echo function. If PE router 10B determines, for example, thatlink 16E has failed, PE router 10B may stop sending BFD packets inresponse to the BFD packets sent by PE router 10C. For instance PErouter 10C may send to PE router 10B, BFD packets that include at leastthe local discriminator for BFD session 20B and an EVPN service labelthat was advertised by PE router 10B. In response to sending the BFDpackets that include the local discriminator and the EVPN service label,PE router 10C may receive from PE router 10B, BFD packets that includethe remote discriminator generated PE router 10B, the localdiscriminator generated by PE router 10C and an EVPN service label thatwas advertised by PE router 10C to PE router 10B.

As described in FIG. 1, PE router 10C may install respective forwardingnext hops for each of PE routers 10A and 10B in forwarding structures(e.g., forwarding tables, context tables, radix trees, next hop lists,chained next hops, or the like) of one or more of PE router 10C'sforwarding units to forward network packets. For instance, RPD 99 maystore respective forwarding next hops for each of PE routers 10A and 10Bin forwarding information 106. The forwarding next hop for PE router 10Amay specify one or more operations that cause forwarding engine 86 toforward a network packet out an egress interface, such as interface 88A,that couples PE router 10C to PE router 10A. Similarly, a forwardingnext hop for PE router 10B may cause forwarding engine 86 to forward anetwork packet out an egress interface, such as interface 88B, thatcouples PE router 10C to PE router 10B.

Forwarding engine 86 may initially use the forwarding next hops to loadbalance traffic received at PE router 10C between PE routers in EthernetSegment 14, such as PE router 10A and PE router 10B. This forwarding oftraffic may occur while BFD session 20B is executing between PE routers10B and 10C. When forwarding traffic, forwarding engine 86 may perform alookup on packet header information of a packet and determine that thepacket is destined for a device in customer network 6B. For example, alookup on a packet header may resolve to an entry in a forwardingstructure of forwarding information 106 such as a radix tree thatcorresponds to one or more forwarding next hops. Forwarding engine 86may implement a hashing algorithm to vary the selection of theforwarding next hops for PE routers 10A and 10B in order to load-balancenetwork packets between PE routers 10A and 10B, which are each includedin Ethernet Segment 14. Forwarding engine 86 may initially load-balancethe forwarding of network packets to PE routers 10A and 10B while nofailure has occurred at communication link 16E and no failure hasoccurred at a network device or a link on a path between route reflector18 and PE router 10B.

At a later time, link 16E may have failed or network traffic is nolonger flowing from PE router 10C to CE router 8B via PE router 10B. PErouter 10B, for instance, may have determined that link 16E has failed.As such, PE router 10B may stop replying to PE router 10C with BFDpackets that include the local and remote descriptors corresponding toBFD session 20B for Ethernet Segment 14 (having ESI 200) between PErouters 10B and 10C. BFD module 107 of PE router 10C may determine thatBFD session 20B has been interrupted or terminated because BFD module107 has not received BFD packet within a threshold period of time fromPE router 10B that includes the remote discriminator of PE router 10Band local discriminator generated by PE router 10C. Accordingly, BFDmodule 107 may determine that network traffic is no longer able to flowfrom PE router 10C to CE router 8B via PE router 10B. BFD module 107 mayinvalidate a forwarding next hop for PE router 10B in forwardinginformation 106, as described in FIG. 1.

Upon invalidating the forwarding next hop for PE router 10B, ifforwarding engine 86 performs a lookup on a packet destined for a devicein customer network 6B using forwarding information 104, forwardingengine 86 may forward the network packet using the operations of theforwarding next hop for PE router 10A. Invalidating the forwarding nexthop for PE router 10B may remove a route in forwarding information 106for PE router 10B, such that traffic is no longer forwarded byforwarding engine 86 to PE router 10B. In this way, by detecting at BFDmodule 107 that BFD session 20B between PE router 10C and PE router 10Bfor ESI 200 has been terminated or interrupted, BFD module 107 may causeforwarding engine 86 to stop load-balancing network traffic to PE router10B, thereby reducing the number of packets dropped at PE router 10B orbetween route reflector 18 and PE router 10B after a failure of link16E. By using BFD sessions on a per-ESI basis between PE routers 10A and10B, BFD module 107 may detect a link failure more quickly in the dataplane, than if PE router 10C eventually received a BGP message in thecontrol plane that indicated a route no longer existed between PE router10B and CE router 8B.

The architecture of PE router 10C illustrated in FIG. 2 is shown forexemplary purposes only. The invention is not limited to thisarchitecture. In other examples, PE router 10C may be configured in avariety of ways. In one example, some of the functionally of controlunit 82 may be distributed within IFCs 88. In another example, controlunit 82 may comprise a plurality of packet forwarding engines operatedas slave routers.

Control unit 82 may be implemented solely in software, or hardware, ormay be implemented as a combination of software, hardware, or firmware.For example, control unit 82 may include one or more processors whichexecute software instructions. In that case, the various softwaremodules of control unit 82 may comprise executable instructions storedon a computer-readable medium, such as computer memory or hard disk.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof. Various featuresdescribed as modules, units or components may be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices or other hardware devices. In some cases, various featuresof electronic circuitry may be implemented as one or more integratedcircuit devices, such as an integrated circuit chip or chipset.

If implemented in hardware, this disclosure may be directed to anapparatus such a processor or an integrated circuit device, such as anintegrated circuit chip or chipset. Alternatively or additionally, ifimplemented in software or firmware, the techniques may be realized atleast in part by a computer-readable data storage medium comprisinginstructions that, when executed, cause a processor to perform one ormore of the methods described above. For example, the computer-readabledata storage medium may store such instructions for execution by aprocessor.

A computer-readable medium may form part of a computer program product,which may include packaging materials. A computer-readable medium maycomprise a computer data storage medium such as random access memory(RAM), read-only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),Flash memory, magnetic or optical data storage media, and the like. Insome examples, an article of manufacture may comprise one or morecomputer-readable storage media.

In some examples, the computer-readable storage media may comprisenon-transitory media. The term “non-transitory” may indicate that thestorage medium is not embodied in a carrier wave or a propagated signal.In certain examples, a non-transitory storage medium may store data thatcan, over time, change (e.g., in RAM or cache).

The code or instructions may be software and/or firmware executed byprocessing circuitry including one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, functionality described in this disclosure may be providedwithin software modules or hardware modules.

FIG. 3 is a flowchart illustrating example operations of multiplenetwork devices in accordance with techniques of the disclosure. Exampleoperations in accordance with techniques of the disclosure areillustrated for example purposes with respect to PE routers 10A-10C ofFIGS. 1-2. For simplicity, operation of other network devices is notshown in FIG. 4.

As shown in FIG. 3, the PE routers of the Ethernet segment, e.g., PErouters 10A-10C of Ethernet segment 14, may, at initial configurationand startup, exchange Ethernet AD routes. For instance, each of PErouters 10A and 10B may send Ethernet AD routes to PE router 10C (150,154). PE router 10C may listen for Ethernet AD routes and receivepackets specifying Ethernet AD routes from each of PE routers 10A and10B.

PE router 10C may determine that each ESI included in the respectiveEthernet AD routes from PE routers 10A and 10B is the same (152). Forinstance, PE router 10C may compare the ESIs of the respective EthernetAD routes and determines that the ESIs match one another. PE router 10Cmay also determine that PE router 10C is coupled to PE routers 10A and10B and that PE routers 10A and 10B are each configured in active-activemode in the same Ethernet Segment. PE router 10C may send ESI Pingrequests to PE routers 10A and 10B (158). With respect to the ESI Pingrequest packet for PE router 10B, PE router 10C may generate the ESIPing request packet as an LSP ping exploration packet for PE router 10Bwith an attached EVPN label stack that includes an outer transport labelfor a next hop on a path to PE router 10B and inner service label thatrepresents the MPLS Alias Label, which PE router 10B previouslyadvertised to PE router 10C in an Ethernet AD route.

Each of PE routers 10A and 10B may receive the ESI Ping requests. Basedon determining that the message includes an EVPN label stack of innerservice label and outer transport label, and further that the contentsof the message includes a BFD local discriminator, PE routers 10A and10B may generate ESI Ping reply messages. As described in FIG. 1, theESI Ping reply message may be an LSP-Ping exploration reply packet thatfurther includes a label stack with an outer transport label for a nexthop in a path from PE router 10B to PE router 10C, and inner servicelabel that represents the MPLS Alias Label, which PE router 10Cpreviously advertised to PE router 10B in an Ethernet AD route. The ESIPing reply generated by PE router 10B in response to the ESI Pingrequest packet may include the BFD local discriminator received from PErouter 10B and a BFD remote discriminator (or “Your Discriminator”) thatcorresponds to or identifies PE router 10B as a participant in theparticular BFD session. PE router 10B may send an ESI Ping reply back toPE router 10C (160). Similarly, PE router 10A may send an ESI Ping replyback to PE router 10C (158).

PE router 10C may initiate and establish BFD session with PE routers 10Aand 10B by exchanging BFD packets in separate per-ESI BFD sessions witheach of PE routers 10A and 10B, where the BFD packets include therespective local and remote discriminators for each respective BFDsession (162). The BFD packets may include an EVPN label stack having anouter transport label and inner service label to route the packetsbetween PE routers for a specific Ethernet Segment, such as ESI 200. Inthis way, a BFD session may be established on a per-ESI basis becausethe BFD packets have EVPN label stack that includes an inner servicelabel that corresponds to a particular ESI for an Ethernet Segment.

PE router 10C may forward network traffic to Ethernet Segment 14 havingESI 200 by load-balancing network traffic to each of PE routers 10A and10B (166). PE routers 10A and 10B receive the network traffic (164, 168)and forward the network traffic to customer network 6B or otherrespective destination. At a later time, communication link 16E, asshown in FIG. 1, may fail. PE router 10C may detect that the BFD session20B has terminated or been interrupted and therefore network traffic canno longer flow from PE router 10C to CE router 8B via a path thatincludes PE router 10B (170). For instance, PE router 10B stop sendingBFD reply messages for BFD session 20B that corresponds to ESI 200because PE router 10B cannot forward network packets to CE router 8B forESI 200.

PE router 10C may update its forwarding state to re-direct networktraffic for Ethernet Segment 14 to other PE routers that are alsooperating in active-active mode for Ethernet Segment 14, such as PErouter 10A (172). For instance, to re-direct network traffic, PE router10C may invalidate a forwarding next hop for PE router 10B in itsforwarding information. Accordingly, if PE router 10C performs a lookupon a packet destined for a device in customer network 6B, PE router 10Cmay forward the network packet using the operations of the forwardingnext hop for PE router 10A (174). PE router 10C may forward traffic forEthernet Segment 14 to PE router 10A and bypass PE router 10B altogetherbecause PE router 10B cannot forward traffic to customer network 6B dueto the link failure. PE router 10A may receive the network traffic andforward it to customer network 6B or other intended destination.

In some examples, the end-to-end convergence time for a network failuremay be the sum of the time taken to detect the failure and the timetaken for corrective action. Techniques of the disclosure may reduce theupper bound on time taken to detect failure by leveraging data plane OAMmechanisms, which are independent of load in the control plane and othernetwork events. The techniques may also ensure that local repair isexercised to ensure faster convergence. Accordingly, techniques of thedisclosure may reduce the end to end convergence in multi-homingtopologies and may make end to end convergence independent of the scaleof the control plane.

FIGS. 4A-4B are block diagrams illustrating an exemplary format of anESI Ping request packet and EVPN ESI FEC information, in accordance withtechniques of the disclosure. ESI Ping request packet 200 includes apacket header 202 and a packet body 204. In the example of FIG. 4A, PErouter 10C may generate the ESI Ping request packet.

As illustrated in FIG. 4A, packet header 202 may include data fields forEVPN and Internet Protocol v4 or v6. For instance, packet header 202includes an MPLS Header Transport Label 206 and an MPLS Header AliasingLabel 208. In the example of FIG. 4A, if PE router 10C generates ESIPing request packet 200 for PE router 10B, MPLS Header Transport Label206 may corresponds to a label-switched path between PE router 10C and anext hop network device en route to PE router 10B. For instance, thenext hop network device may have previously advertised the MPLS HeaderTransport Label 206 to PE router 10C, which PE router 10C may apply topackets that are sent to the next hope network device. MPLS HeaderAliasing Label 208 may be the MPLS Alias Label that PE router 10Bpreviously advertised in its Ethernet AD route to PE router 10C. Packetheader 202 may also include IP header 210. IP header 210 may include butis not limited to: IP source address, IP destination address, sourceport, destination port, and hop limit (or time-to-live value).

Packet body 204 may include a User Datagram Protocol (UDP) packet.Packet body 204 may include a UDP header 212. UDP header 212 may includebut is not limited to: a source port, destination port, length, andchecksum. Packet body 204 may include MPLS ECHO request information 214as specified in RFC 4379 “Detecting Multi-Protocol Label Switched (MPLS)Data Plane Failures,” February 2006, the entire contents of which areincorporated herein by reference. Packet body 204 may also includeTarget FEC Stack (EVPN ESI FEC) 216. The contents of Target FEC Stack216 are further illustrated in FIG. 4B. Packet body 204 may also includea BFD Discriminator TLV 218. BFD Discriminator TLV 218 may specify alocal (“My”) discriminator generated by PE 10C.

FIG. 4B illustrates the contents of Target FEC Stack (EVPN ESI FEC) 216of FIG. 4A in further detail as Target FEC Stack (EVPN ESI FEC) 220. Forinstance, Target FEC Stack (EVPN ESI FEC) 220 includes a routedistinguisher that corresponds to PE router 10C. Target FEC Stack (EVPNESI FEC) 220 may also include the Ethernet Segment Identifier for theper-ESI BFD session, such as ESI 200 in FIG. 1. In some examples, the 16highest order bits of Target FEC Stack (EVPN ESI FEC), which are labeledas “zero value”, must be zero.

With respect to techniques of this disclosure, MAC addresses of devicesmay be learned by PE routers in EVPN over ESI and ESI Ping packets usedon demand may provide a tool to verify an ESI advertised by remote PE.ESI Ping packets used on demand may help validating control plane anddata plane synchronization.

FIG. 5 is a block diagram illustrating an exemplary format of BFD packet300 for a per-ESI BFD session, in accordance with techniques of thedisclosure. BFD packet 300 includes a packet header 302 and a packetbody 304. In the example of FIG. 5, PE router 10C may generate BFDpacket 300.

As illustrated in FIG. 5, packet header 302 may include data fields forEVPN and Internet Protocol v4 or v6. For instance, packet header 302includes an MPLS Header Transport Label 306 and an MPLS Header AliasingLabel 308. In the example of FIG. 5, if PE router 10C generates BFDpacket 300 for PE router 10B, MPLS Header Transport Label 306 maycorresponds to a label-switched path between PE router 10C and a nexthop network device that is one hop downstream from PE router 10C enroute to PE router 10B. For instance, the next hop network device mayhave previously advertised the MPLS Header Transport Label 306 to PErouter 10C, which PE router 10C may apply to packets that are sent tothe next hope network device. MPLS Header Aliasing Label 308 may be theMPLS Alias Label that PE router 10B previously advertised in itsEthernet AD route to PE router 10C. Packet header 302 may also includeIP header 310. IP header 310 may include but is not limited to: IPsource address, IP destination address, source port, destination port,and hop limit (or time-to-live value). Packet body 304 may include butis not limited to the contents of a BFD packet as described in RFC 5880,as shown in FIG. 5. Packet body 304 may include additional information,such as UDP header information and/or a BFD Authentication Section.

In various examples of this disclosure, BFD packets may be sent on thesame tunnel as that of customer traffic for the Ethernet Segment. BFDpackets may have dentation IP addresses set to 127/8 ranges as explainedin RFC 5884. BFD packet may have source IP address set to the outgoinginterface IP address. Bootstrapping may be used to associate a BFDpacket to BFD session if the BFD packet carries destination address IPset to 127/8 and also the inner label is the same for ESI in an instancefor the BFD session. In other words, ESI Ping may be used to bootstrap aBFD session, to exchange BFD discriminator as defined in RFC 5884, suchthat BFD packet will be using same MPLS header as that of data packetsto reach egress PE router.

Techniques of the disclosure using per-ESI BFD session may providedetection of failures in the data plane as the techniques may not notdepend on sending a withdrawal message to a route reflector or otherintermediate network devices and may not depend on other events in thenetwork and load on the network. Techniques of the disclosure mayprovide faster repair because the techniques may trigger BFD-based localrepair and later global repair. Techniques of the disclosure usingperiodic ESI ping can also check the control plane and data planesynchronization and can detect traffic black-holing. BFD triggered localrepair using techniques of the disclosure may reduce the time taken forcorrective action irrespective of scale.

Various embodiments of the invention have been described. These andother embodiments are within the scope of the following claims.

1. A method comprising: determining, by a remote provider edge (PE)router coupled by an intermediate network to at least two PE routersoperating in an Ethernet Virtual Private Network (EVPN) with the remotePE router, that the at least two PE routers are locally connected to amulti-homed customer network by a particular Ethernet Segment; andsending, by the remote PE router and based on determining that the atleast two PE routers operating in the EVPN are connected to themulti-homed customer network by the particular Ethernet Segment, anEthernet Segment Identifier (ESI) Ping request packet through theintermediate network to one of the at least two PE routers, wherein theESI Ping request packet includes at least a Bidirectional ForwardingDetection (BFD) discriminator and an ESI for the particular EthernetSegment that locally connects the at least two PE routers to themulti-homed customer network.
 2. The method of claim 1, furthercomprising: in response to receiving a reply packet from the one of theat least two PE routers, executing, by the remote PE router and with theone of the at least two PE routers, a BFD session for the ESI throughthe EVPN; and in response to determining that the BFD session with theone of the at least two PE routers for the ESI has at least beeninterrupted or terminated, forwarding network traffic through the EVPNto bypass the one of the at least two PE routers.
 3. The method of claim2, wherein forwarding network traffic through the EVPN to bypass the oneof the at least two PE routers further comprises: updating, by theremote PE router, a forwarding unit of the remote PE router to onlyencapsulate network traffic with an EVPN label stack for the remainingones of the at least two PE routers.
 4. The method of claim 2, whereinthe one of the at least two PE routers is a first PE router and anotherof the at least two PE routers is a second PE router, the method furthercomprising: installing, by the remote PE router and within at least oneforwarding unit of the remote PE router, a first forwarding next hopthat forwards network traffic to the first PE router and a secondforwarding next hop that forwards network traffic to the second PErouter; wherein load-balancing network traffic from the remote PE routerto the at least two PE routers further comprises, forwarding networktraffic between the at least two PE routers based at least in part onthe first forwarding next hop and the second forwarding next hop; and inresponse to determining that the BFD session with the one of the atleast two PE routers for the ESI has at least been interrupted orterminated, invalidating the first forwarding next hop, such thatnetwork traffic is forwarded without first forwarding next hop.
 5. Themethod of claim 2, further comprising: prior to determining that the atleast two PE routers are included in a particular Ethernet Segment,receiving, by the remote PE router and from the one of the at least twoPE routers, an Ethernet Auto-Discovery route that includes the ESI; andafter invalidating the first forwarding next hop, receiving, by theremote PE router and from the one of the at least two PE routers, awithdrawal of the Ethernet Auto-Discovery route from the one of the atleast two PE routers.
 6. The method of claim 2, wherein determining thatthe BFD session has at least been interrupted or terminated comprises:determining that the remote PE router has not received, within athreshold period of time, a BFD packet that: includes the remotediscriminator, and is sent by the one of the at least two PE routers inresponse to the BFD packet that includes the local discriminator.
 7. Themethod of claim 1, wherein the BFD discriminator is a localdiscriminator, wherein executing, by the remote PE router and with theone of the at least two PE routers, the BFD session for the ESI furthercomprises: sending, by the remote PE router and to the one of the atleast two PE routers, BFD packets that include at least the localdiscriminator and an EVPN service label that was advertised by the oneof the at least two PE routers to the remote PE router; and in responseto sending the BFD packets that include at least the local discriminatorand the EVPN service label, receiving, by the remote PE router and fromthe one of the at least two PE routers, BFD packets that include atleast a remote discriminator generated by the one of the at least two PErouters and an EVPN service label that was advertised by the remote PErouter to the one of the at least two PE routers.
 8. The method of claim1, wherein the one of the at least two PE routers is a first PE router,wherein determining that the BFD session has at least been interruptedor terminated is based at least in part on the first PE routerinterrupting or terminating the BFD session with the remote PE router inresponse to the first PE router determining a communication link for theEthernet Segment has failed between the first PE router and a customeredge router.
 9. The method of claim 1, wherein the BFD discriminator isa local discriminator, and wherein the reply back is an ESI Ping replypacket that includes at least a remote discriminator generated by theone of the at least two PE routers and the local discriminator generatedby the remote PE router.
 10. The method of claim 1, wherein the one ofthe at least two PE routers is a first PE router, wherein the ESI Pingrequest is a first ESI Ping request, the method further comprising:sending a second ESI Ping request packet through the intermediatenetwork to a second PE router of the at least two PE routers; inresponse to receiving a second reply packet from the second PE router ofthe at least two PE routers, executing, by the remote PE router and withthe second PE router, a second BFD session for the ESI through the EVPN.11. A remote PE router coupled by an intermediate network to at leasttwo PE routers operating in an Ethernet Virtual Private Network (EVPN)with the remote PE router, wherein the remote PE router comprises: arouting engine that determines that the at least two PE routers arelocally connected to a multi-homed customer network by a particularEthernet Segment; and a forwarding unit that sends, based on the routingengine determining that the at least two PE routers operating in theEVPN are connected to the multi-homed customer network by the particularEthernet Segment, an Ethernet Segment Identifier (ESI) Ping requestpacket through the intermediate network to one of the at least two PErouters, wherein the ESI Ping request packet includes at least aBidirectional Forwarding Detection (BFD) discriminator and an ESI forthe particular Ethernet Segment that locally connects the at least twoPE routers to the multi-homed customer network.
 12. The remote PE routerof claim 11, wherein the forwarding unit, in response to the remote PErouter receiving a reply packet from the one of the at least two PErouters, executes with the one of the at least two PE routers, a BFDsession for the ESI through the EVPN; and wherein the forwarding unit,in response to determining that the BFD session with the one of the atleast two PE routers for the ESI has at least been interrupted orterminated, forwards network traffic through the EVPN to bypass the oneof the at least two PE routers.
 13. The remote PE router of claim 12,wherein routing engine updates the forwarding unit to only encapsulatenetwork traffic with an EVPN label stack for the remaining ones of theat least two PE routers.
 14. The remote PE router of claim 12, whereinthe one of the at least two PE routers is a first PE router and anotherof the at least two PE routers is a second PE router, wherein theforwarding engine installs, within the forwarding unit of the remote PErouter, a first forwarding next hop that forwards network traffic to thefirst PE router and a second forwarding next hop that forwards networktraffic to the second PE router; wherein the forwarding unit forwardsnetwork traffic between the at least two PE routers based at least inpart on the first forwarding next hop and the second forwarding nexthop; and wherein the forwarding unit, in response to determining thatthe BFD session with the one of the at least two PE routers for the ESIhas at least been interrupted or terminated, invalidates the firstforwarding next hop, such that network traffic is forwarded withoutfirst forwarding next hop.
 15. The remote PE router of claim 12, whereinthe routing engine, prior to determining that the at least two PErouters are included in a particular Ethernet Segment, receives, fromthe one of the at least two PE routers, an Ethernet Auto-Discovery routethat includes the ESI; and wherein the routing engine, after the forwardunit has invalidated the first forwarding next hop, receives, from theone of the at least two PE routers, a withdrawal of the EthernetAuto-Discovery route from the one of the at least two PE routers. 16.The remote PE router of claim 12, wherein the forwarding unit determinesthat the remote PE router has not received, within a threshold period oftime, a BFD packet that: includes the remote discriminator, and is sentby the one of the at least two PE routers in response to the BFD packetthat includes the local discriminator.
 17. The remote PE router of claim11, wherein the BFD discriminator is a local discriminator, wherein theforwarding unit sends, to the one of the at least two PE routers, BFDpackets that include at least the local discriminator and an EVPNservice label that was advertised by the one of the at least two PErouters to the remote PE router; and wherein the forwarding unit, inresponse to sending the BFD packets that include at least the localdiscriminator and the EVPN service label, receives, from the one of theat least two PE routers, BFD packets that include at least a remotediscriminator generated by the one of the at least two PE routers and anEVPN service label that was advertised by the remote PE router to theone of the at least two PE routers.
 18. The remote PE router of claim11, wherein the one of the at least two PE routers is a first PE router,wherein the interruption or termination of the BFD session is based atleast in part on the first PE router interrupting or terminating the BFDsession with the remote PE router in response to the first PE routerdetermining a communication link for the Ethernet Segment has failedbetween the first PE router and a customer edge router.
 19. The remotePE router of claim 11, wherein the BFD discriminator is a localdiscriminator, and wherein the reply back is an ESI Ping reply packetthat includes at least a remote discriminator generated by the one ofthe at least two PE routers and the local discriminator generated by theremote PE router.
 20. A computer-readable medium comprising instructionsfor causing at least one programmable processor of a remote provideredge (PE) router, to: determine, by the remote provider edge (PE) routercoupled by an intermediate network to at least two PE routers operatingin an Ethernet Virtual Private Network (EVPN) with the remote PE router,that the at least two PE routers are locally connected to a multi-homedcustomer network by a particular Ethernet Segment; and send, based ondetermining that the at least two PE routers operating in the EVPN areconnected to the multi-homed customer network by the particular EthernetSegment, an Ethernet Segment Identifier (ESI) Ping request packetthrough the intermediate network to one of the at least two PE routers,wherein the ESI Ping request packet includes at least a BidirectionalForwarding Detection (BFD) discriminator and an ESI for the particularEthernet Segment that locally connects the at least two PE routers tothe multi-homed customer network.